1. Field of the Invention
The present invention relates to an image processing apparatus capable of generating virtual image data which is obtained when a light ray incident on an imaging apparatus is captured on a virtual position of an image sensor and an image processing method.
2. Description of the Related Art
There is a known technique for acquiring virtual image data called light field photography technique (R. NG “Fourier Slice Photography” (ACM SIGGRAPH 2005, 2005) hereinafter referred to as a non-patent literature 1). According to the light field photography technique, an image of an object is taken by using a light ray incident on an imaging apparatus (e.g., a camera) by an image sensor set at a virtual position which is different from an actual position. Regarding the light field photography technique, a path of a light ray (a light ray path) emitted the from an object and incident on the imaging apparatus is described as a straight line that passes through two points on two different planes in a three-dimensional space. Further, one light ray (or a group of light rays passing through an approximately the same light ray path and having an approximately the same light quantity) is considered as including a certain light quantity. According to the present invention, light ray information which is a combination of routing information about the light ray path and radiance information about the light quantity is called light field data (LF data). More specifically, the LF data is a function of four parameters of the light quantity of one light ray (four parameters that define each point in two planes). When a term light field (LF) is used, it means a “field” described by a plurality of LF data pieces.
Based on the LF data, the virtual image data is acquired. The virtual image data is obtained at a virtual position which is different from the actual position of the image sensor of the imaging apparatus, when imaging is performed using a light ray from an object and incident on the imaging apparatus. Acquisition of such virtual image data is discussed in the non-patent literature 1.
There are two main methods for acquiring the LF data. One is a method that acquires the LF data by arranging a micro lens array in front of the image sensor. The other is a method that controls an aperture position of a diaphragm and acquires the LF data (C. K. Liang, et al. “Programmable Aperture Photography: Multipixeled Light Field Acquisition” hereinafter referred to as a non-patent literature 2). In principle, when the virtual image data is generated based on the LF data obtained by arranging a micro lens array in front of the image sensor, the resolution of the virtual image given by the virtual image data will be lower compared to the resolution of the image sensor.
Further, according to the non-patent literature 2, the LF data is acquired using a variable aperture referred to as a programmable aperture. From the obtained LF data, a distance map whose resolution is same as the resolution of the image sensor is estimated, and high-quality image is generated by view interpolation using the distance map. The distance map is generated based on feature points of the image. Thus, it is not possible to generate an accurate distance map when the object has few edges or the object is substantially flat. Thus, according to the method discussed in the non-patent literature 2, it is substantially impossible to acquire virtual image data which is obtained when the imaging is performed by setting the image sensor at a virtual position with high resolution.